Kernel Recursive Least Squares

We present a non-linear kernel-based version of the Recursive Least Squares (RLS) algorithm. Our Kernel-RLS algorithm performs linear regression in the feature space induced by a Mercer kernel, and can therefore be used to recursively construct the minimum meansquared-error regressor. Sparsity (and therefore regularization) of the solution is achieved by an explicit greedy sparsification process that admits into the kernel representation a new input sample only if its feature space image is linearly independent of the images of previously admitted samples. Most importantly, this sparsification procedure allows the algorithm to operate online. We demonstrate the performance and scaling properties of the Kernel-RLS algorithm as compared to a state-of-the-art Support Vector Regression algorithm, on both synthetic and real data.